Image processing apparatus, method for controlling same, imaging apparatus, and program

ABSTRACT

An image processing apparatus includes a unit configured to read an image file including viewpoint images, a unit configured to determine a photographic condition, an image processing unit configured to apply image processing to image data based on the viewpoint images, and a limitation unit configured to limit a settable parameter of the image processing based on the photographic condition.

BACKGROUND Field of the Disclosure

The present disclosure relates to an image processing technique usingviewpoint images.

Description of the Related Art

Several types of image processing have been discussed which use aplurality of viewpoint images including the same object image obtainedby photographing. Japanese Patent Application Laid-Open No. 2012-186790discusses a technique for generating an image from an arbitrary(virtual) point of view out of images captured at more than one point ofview while adjusting a composition ratio between the images. JapanesePatent Application Laid-Open No. 2014-228586 discusses a technique forrelatively shifting and compositing a plurality of parallax images togenerate an image refocused on a virtual image plane. The parallaximages are obtained from an image sensor in which a plurality ofphotoelectric conversion units is assigned to a single microlens.

When such image processing using a plurality of viewpoint images isperformed in an application based on user operations, it is difficult tomake a setting because a change in a parameter can cause differentdegrees of image effect depending on a photographic condition.

SUMMARY

The present disclosure is directed to providing an image processingapparatus for performing image processing using a plurality of viewpointimages and a method for controlling the same, wherein a desired imageeffect is obtained by a user's simple operations.

According to an aspect of the present disclosure, an image processingapparatus includes an acquisition unit configured to obtain a pluralityof viewpoint images and imaging information corresponding to theplurality of viewpoint images, an image processing unit configured toapply image processing to image data based on the viewpoint images, asetting unit configured to set a parameter of the image processing bythe image processing unit based on a user operation, and a limitationunit configured to limit a parameter set table by a user via the settingunit in the image processing based on the imaging information.

According to another aspect of the present disclosure, a method forcontrolling an image processing apparatus includes obtaining a pluralityof viewpoint images and imaging information corresponding to theplurality of viewpoint images, imposing a limitation on a user-settableparameter in image processing by an image processing unit based on theimaging information, setting a parameter of the image processing by theimage processing unit based on a user operation and the limitation, andapplying the image processing by the image processing unit to image databased on the viewpoint images.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an imageprocessing apparatus according to one or more aspects of the presentdisclosure.

FIG. 2 is a flowchart of processing for adjusting resolution feelingaccording to one or more aspects of the present disclosure.

FIG. 3 is a graph illustrating an effect of modification processing fordividing viewpoint images according to one or more aspects of thepresent disclosure.

FIG. 4 is a conceptual diagram for describing refocus processingaccording to one or more aspects of the present disclosure.

FIG. 5A is a graphical user interface (GUI) of an application accordingto one or more aspects of the present disclosure. FIG. 5B is a flowchartaccording to one or more aspects of the present disclosure.

FIGS. 6A and 6B are diagrams illustrating tables showing acorrespondence between photographic conditions and image processingaccording to one or more aspects of the present disclosure.

DESCRIPTION OF THE EMBODIMENTS

An exemplary embodiment of the present disclosure will be described indetail below with reference to the drawings.

FIG. 1 is a block diagram illustrating a schematic configuration of animage processing apparatus 100 according to a first exemplary embodimentof the present disclosure. The image processing apparatus 100 includes acontrol unit (central processing unit (CPU)) 101, a signal processingunit 102, a random access memory (RAM) 103, an external storage device104, an operation unit 105, a display unit 106, and a read-only memory(ROM) 108. Such components are communicably connected with each othervia a bus 107.

The control unit 101 includes a CPU. The control unit 101 loads variousprograms stored in the ROM 108 into the RAM 103 and controls anoperation of various parts of the image processing apparatus 100 toperform an overall control on the image processing apparatus 100.

The signal processing unit 102 performs general image processing andspecific image processing on image data loaded into RAM 103 according tothe present exemplary embodiment. The general image processing isperformed on image data that is read from the external storage device104 and loaded into the RAM 103. Examples of the general imageprocessing are synchronization of Bayer-array image signals output froman image sensor, various types of correction processing such as shadingcorrection, white balancing, development processing such as gammacorrection and color conversion processing, encoding, decoding, andcompression processing. The processing specific to the present exemplaryembodiment includes processing for adjusting the resolution feeling andprocessing for controlling a composition ratio of a plurality ofviewpoint images. The resolution feeling adjustment processing includesadjusting the position where the plurality of viewpoint images appearsto be in focus, and adjusting overall resolution feeling. The processingfor controlling the composition ratio of the plurality of viewpointimages is blur shift processing as well as ghost reduction processing.The blur shift processing controls the composition ratio to change apoint of view and shift a blur. The ghost reduction processing reduces aghost occurring in an image. Details of the processing will be describedbelow.

As an example of the RAM 103, a dynamic random access memory (DRAM) canbe used. The RAM 103 includes an area for loading the programs to beexecuted by the control unit 101, and stores various types of arithmeticdata calculated by the control unit 101. An example of the externalstorage device 104 is a nonvolatile memory such as a flash memory. Theexternal storage device 104 stores image data and various parametersneeded for the operation of the control unit 101. The ROM 108 storesvarious programs such as a control program and an application program tobe executed by the control unit 101.

The operation unit 105 includes input devices such as a mouse, akeyboard, and a touch panel. The operation unit 105 instructs thecontrol unit 101 to execute processing desired by a user according to auser's operation. An example of the display unit 106 is a liquid crystaldisplay (LCD). The display unit 106 displays image data and a graphicaluser interface (GUI). A display control of the display unit 106 isperformed by the control unit 101. The bus 107 includes an address bus,a data bus, and a control bus. The bus 107 enables communicationsbetween the components.

The viewpoint images to be read from, the external storage device 104and processed in the present exemplary embodiment are captured by usinga two-dimensional image sensor in which one microlens and segmentedphotoelectric conversion elements (photoelectric conversion units) areprovided for each pixel. The plurality of the segmented photoelectricconversion elements is configured to receive light in different areas ofan exit pupil of an imaging lens via the one microlens to make a pupildivision. Image signals of the viewpoint images of which light isreceived by the plurality of split photoelectric conversion units can beadded to generate a normal imaging signal corresponding to the entireexit pupil of the imaging lens. The image signals of the plurality ofviewpoint images with respect to the captured image are equivalent tolight field data. The light field data is information with respect to aspatial distribution and an angle distribution of light intensity. Thatis, the method for obtaining the plurality of viewpoint images is notlimited thereto. A multi-lens imaging apparatus may be used to obtain aplurality of viewpoint images. An identical single-lens image sensor maybe used to capture the viewpoint images from plural points of view aplurality of times in a time series manner.

Next, the processing for adjusting resolution feeling will be described.The processing is specific to the present exemplary embodiment andperformed by the signal processing unit 102. The processing foradjusting the resolution feeling includes refocus processing andsharpness/unsharpness processing. In the refocus processing, viewpointimages are relatively shifted and synthesized to correct a focusposition. The sharpness/unsharpness processing applies sharpness orunsharpness based on a parallax (image displacement amount) between theviewpoint images.

FIG. 2 illustrates a flowchart of the resolution feeling adjustmentprocessing. The resolution feeling adjustment processing is performed bythe control unit 101 or by various components according to instructionsfrom the control unit 101.

In step S1, the control unit 101 reads a plurality of viewpoint imagesstored in the external storage device 104.

[Image Displacement Amount Distribution]

In the present exemplary embodiment, a distribution of imagedisplacement amounts between the viewpoint images is calculated toperform more effective refocus processing and determine an area to applysharpness/unsharpness to.

In step S2, the control unit 101 performs one-dimensional band-passfilter processing on a first viewpoint luminance signal Y1 in a pupildivision direction (column direction) to generate a first focusdetection signal dYA. The first viewpoint luminance signal Y1 isgenerated from a first viewpoint image I₁ which is Bayer-array red,green, and blue (RGB) signals. The control unit 101 also performs theone-dimensional band-pass filter processing on a second viewpointluminance signal Y2 in a parallactic direction (pupil divisiondirection, column direction) to generate a second focus detection signaldYB. The second viewpoint luminance signal Y2 is generated from a secondviewpoint signal I₂. For example, a first-order differential filter [1,5, 8, 8, 8, 8, 5, 1, −1, −5, −8, −8, −8, −8, −5, −1] may be used as aone-dimensional band-pass filter. The control unit 101 may adjust thepass band of the one-dimensional band-pass filter if needed.

In step S3, the control unit 101 relatively shifts the first and secondfocus detection signals dYA and dYB in the parallactic direction (columndirection) and calculates a correlation amount indicating a degree ofcoincidence of the first and second focus detection signals dYA and dYB.Suppose, for example, a predetermined value is 0.2. The control unit 101generates an image displacement amount distribution M_(DIS)(j, i) basedon the correlation amount. Detection of the image displacement amountmay be limited to a high contrast area where there is no perspectivecompetition or occlusion, so that the detection accuracy of the imagedisplacement amount becomes high and the processing is accelerated.

A first focus detection signal dYA that is the j₂-th (−n₂≦j₂≦n₂) in arow direction and the i₂-th (−m₂≦i₂≦m₂) in the pupil division direction,or column direction, with a position (j, i) at the center will bedenoted by dYA (j+j₂, i+i₂). A second focus detection signal dYB that isthe j₂-th in the row direction and the i₂-th in the column directionwill be denoted by dYB (j+j₂, i+i₂). A shift amount will be denoted by s(−n_(s)≦s≦n_(s)). The control unit 101 then calculates a correlationamount COR_(EVEN)(j, i, s) and a correlation amount COR_(ODD)(j, i, s)in each position (j, i) by Eqs. (1A) and (1B), respectively:

$\begin{matrix}{{{COR}_{EVEN}\left( {j,i,s} \right)} = {\sum\limits_{j_{2} = {- n_{2}}}^{n_{2}}{\sum\limits_{i_{2} = {- m_{2}}}^{m_{2}}{{{{dYA}\left( {{j + j_{2}},{i + i_{2} + s}} \right)} - {{dYB}\left( {{j + j_{2}},{i + i_{2} - s}} \right)}}}}}} & \left( {1A} \right) \\{{{COR}_{ODD}\left( {j,i,s} \right)} = {\sum\limits_{j_{2} = {- n_{2}}}^{n_{2}}{\sum\limits_{i_{2} = {- m_{2}}}^{m_{2}}{{{{dYA}\left( {{j + j_{2}},{i + i_{2} + s}} \right)} - {{dYB}\left( {{j + j_{2}},{i + i_{2} - 1 - s}} \right)}}}}}} & \left( {1B} \right)\end{matrix}$

The correlation amount COR_(ODD)(j, i, s) is an amount obtained bychanging the shift amount between the first and second focus detectionsignals dYA and dYB by a half phase of −1 with respect to theCOR_(EVEN)(j, i, s).

The control unit 101 calculates real-valued shift amounts s at which therespective correlation amounts COR_(EVEN)(j, i, s) and COR_(ODD)(j, i,s) have minimum values, by sub pixel calculation. The control unit 101calculates an average of the real-valued shift amounts s and generatesan image displacement amount distribution M_(DIS)(j, i).

The control unit 101 may separately detect contrast of each area, andset M_(DIS)(j, i)=0 in areas excluded from the calculation of the imagedisplacement amount, like areas where there is a large difference incontrast between the two images.

As described above, in the present exemplary embodiment, the imagedisplacement amount distribution M_(DIS)(j, i) is generated from, theplurality of viewpoint images.

If the resolution feeling adjustment processing is performed for thesecond and subsequent times, the generation of the image displacementamount distribution M_(DIS)(j, i) may be omitted to reduce theprocessing time. Therefore, it is desirable that the generated imagedisplacement amount distribution M_(DIS)(j, i) is recorded on arecording medium of the external storage device 104 in association withthe recorded image data.

The image displacement amount distribution M_(DIS)(j, i) may bemultiplied by a conversion coefficient corresponding to the position (j,i), an aperture value of the imaging lens (imaging optical system), andan exit pupil distance, and thereby converted into a defocus amountdistribution indicating a distribution of defocus amounts of an objectwithin the viewpoint images.

[Image Displacement Difference Amount Distribution]

In step S4 of FIG. 2, the control unit 101 generates an imagedisplacement difference amount distribution M_(DIFF)(j, i) from theimage displacement amount distribution M_(DIS)(j, i) and a predeterminedimage displacement amount.

In step S4, the control unit 101 initially sets an image displacementamount to be modified by the refocus processing of the present exemplaryembodiment as a predetermined image displacement amount p.

In step S4, the control unit 101 then calculates the image displacementdifference amount distribution M_(DIFF)(j, i) from the imagedisplacement amount distribution M_(DIS)(j, i), the predetermined imagedisplacement amount p, and a contrast distribution M_(CON)(j, i) by Eq.(2):

$\begin{matrix}{{M_{DIFF}\left( {j,i} \right)} = {\left( {1 - \frac{{{M_{DIS}\left( {j,i} \right)} - p}}{\sigma_{p}}} \right) \times {M_{CON}\left( {j,i} \right)}}} & (2)\end{matrix}$

where σ_(p)>0.

The image displacement difference amount distribution M_(DIFF)(j, i) isa distribution obtained by multiplying a monotonically decreasing linearfunction by an absolute value of a difference between the imagedisplacement amount distribution M_(DIS)(j, i) and the predeterminedimage displacement amount p, or |M_(DIS)(j, i)−p|, and by the contrastdistribution M_(DIFF)(j, i). The image displacement difference amountdistribution M_(DIFF)(j, i) is positive if |M_(DIS)(j, i)−p|<σ_(p). Theimage displacement difference amount distribution M_(DIFF)(j, i) is 0 if|M_(DIS)(j, i)−p|=σ_(p). The image displacement difference amountdistribution M_(DIFF)(j, i) is negative if |M_(DIS)(j, i)−p|>σ_(p).

The control unit 101 sets M_(DIFF)(j, i)=(1−|p|/σ_(p))×M_(CON)(j, i) inareas where the contrast distribution M_(CON)(j, i) has a value smallerthan a predetermined value (for example, 0.2) and that are excluded fromthe calculation of the image displacement amount. Other values may beset if needed.

[Modified Viewpoint Images]

In step S5 of FIG. 2, the control unit 101 performs first sharpening andfirst smoothing processing on the first and second viewpoint images I₁and I₂ (first to N_(LF)-th viewpoint images) according to the imagedisplacement difference amount distribution M_(DIFF)(j, i). The controlunit 101 thereby generates a first modified viewpoint image and a secondmodified viewpoint image (first to N_(LF)-th modified viewpoint images).

In the present exemplary embodiment, in areas where the imagedisplacement difference amount distribution M_(DIFF)(j, i) is 0 orhigher (M_(DIFF)(j, i)≧0), the control unit 101 performs processing forincreasing a difference between the viewpoint images to sharpen theparallax (crosstalk correction processing, first sharpening processing)on the first and second viewpoint images I₁ and I₂ (plurality ofviewpoint images). In areas where the image displacement differenceamount distribution M_(DIFF)(j, i) is below 0 (M_(DIFF)(j, i)<0), thecontrol unit 101 performs processing for reducing a difference betweenviewpoint images to smooth a parallax (crosstalk processing, firstsmoothing processing). By such processing, the control unit 101generates the first modified viewpoint image and the second modifiedviewpoint image (plurality of modified viewpoint images).

In step S5 of FIG. 2, the control unit 101 initially sets a firstintensity parameter k_(ct)≧0 for specifying intensity of the processingfor increasing a difference between the viewpoint images to sharpen theparallax (crosstalk correction processing, first sharpening processing)on the first and second viewpoint images (plurality of viewpointimages). Alternatively, the control unit 101 sets a first intensityparameter k_(ct)≧0 for specifying intensity of the processing forreducing a difference between the viewpoint images to smooth theparallax (crosstalk processing, first smoothing processing).

In step S5, the control unit 101 then sets a first intensity parameterdistribution K_(ct)(j, i) by Eq. (3):

The first intensity parameter distribution K_(ct)(j, i) is proportionalto the image displacement difference amount distribution M_(DIFF)(j, i)with k_(ct) as a proportionality factor.

K _(ct)(j,i)=k _(ct) ×M _(DIFF)(j,i).  (3)

In step S5, the control unit 101 then performs processing of Eqs. (4A)and (4B) on the first and second viewpoint images I₁(j, i) and I₂(j, i)(first to N_(LF)-th viewpoint images). The control unit 101 therebygenerates a first modified viewpoint image MI₁(j, i) and a secondmodified viewpoint image MI₂(j, i) (first to N_(LF)-th modifiedviewpoint images).

$\begin{matrix}{{\begin{pmatrix}{{MI}_{1}\left( {j,i} \right)} \\{{MI}_{2}\left( {j,i} \right)}\end{pmatrix} = {\begin{pmatrix}{1 + {K_{ct}\left( {j,i} \right)}} & {- {K_{ct}\left( {j,i} \right)}} \\{- {K_{ct}\left( {j,i} \right)}} & {1 + {K_{ct}\left( {j,i} \right)}}\end{pmatrix}\begin{pmatrix}{I_{1}\left( {j,i} \right)} \\{I_{2}\left( {j,i} \right)}\end{pmatrix}}},\left( {{K_{ct}\left( {j,i} \right)} \geq 0} \right),} & \left( {4A} \right) \\{{\begin{pmatrix}{{MI}_{1}\left( {j,i} \right)} \\{{MI}_{2}\left( {j,i} \right)}\end{pmatrix} = {\begin{pmatrix}\frac{1 - {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}} & \frac{- {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}} \\\frac{- {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}} & \frac{1 - {K_{ct}\left( {j,i} \right)}}{1 - {2{K_{ct}\left( {j,i} \right)}}}\end{pmatrix}\begin{pmatrix}{I_{1}\left( {j,i} \right)} \\{I_{2}\left( {j,i} \right)}\end{pmatrix}}},\left( {{K_{ct}\left( {j,i} \right)} < 0} \right),} & \left( {4B} \right)\end{matrix}$

Eq. (4A) represents processing for increasing a difference between thefirst and second viewpoint images I₁(j, i) and I₂(j, i) (plurality ofviewpoint images) to sharpen the parallax in areas where the firstintensity parameter distribution K_(ct)(j, i) (image displacementdifference amount distribution) is 0 or above (K_(ct)(j, i)k_(ct)×M_(DIFF)(j, i)≧0). Eq. (4B) represents processing for reducing adifference between the first and second viewpoint images I₁(j, i) andI₂(j, i) (plurality of viewpoint images) to smooth the parallax in areaswhere the first intensity parameter distribution K_(ct)(j, i) (imagedisplacement difference amount distribution) is below 0 (K_(ct)(j,i)=k_(ct)×M_(DIFF)(j, i)<0).

FIG. 3 is a graph illustrating an example of the processing forincreasing a difference between the first and second viewpoint imagesI₁(j, i) and I₂(j, i) (plurality of viewpoint images) to sharpen theparallax (crosstalk correction processing, first sharpening processing).The horizontal axis indicates a pixel position. The vertical axisindicates a pixel value (signal level). In FIG. 3, examples of the firstviewpoint image I₁(j, i) (before modification A) and the secondviewpoint image I₂(j, i) (before modification B) before the sharpeningprocessing (crosstalk correction processing, first sharpeningprocessing) are illustrated by broken lines. Examples of the firstmodified viewpoint image MI₁(j, i) (after modification A) and the secondmodified viewpoint image MI₂(j, i) (after modification B) after thesharpening processing (crosstalk correction processing, first sharpeningprocessing) of Eq. (4A) are illustrated by solid lines. The processingfor increasing a difference between the viewpoint images to sharpen theparallax (crosstalk correction processing, first sharpening processing)enhances portions where there is a large difference between theviewpoint images before the processing. Portions where there is a smalldifference between the viewpoint images are not much changed. Thus, itcan be seen that the parallax between the viewpoint images is sharpened.

The smoothing processing (crosstalk processing, first smoothingprocessing) of Eq. (4B) reduces differences between the first and secondviewpoint images I₁(j, i) and I₂(j, i) (plurality of viewpoint images)to smooth the parallax between the viewpoint images.

As described above, in the present exemplary embodiment, the imageprocessing for sharpening and smoothing according to the contrastdistribution M_(CON)(j, i) and the image displacement amountdistribution M_(DIS)(j, i) is performed on the plurality of viewpointimages. The image processing according to the contrast distributionM_(CON)(j, i) and the image displacement amount distribution M_(DIS)(j,i) may be any of sharpening processing, smoothing processing, and acombination of such processing according to need.

In the present exemplary embodiment, Eqs. (7A), (7B), (2), (3), (4A),and (4B), perform the image processing for sharpening and smoothing onthe viewpoint images more intensely in areas where there is a smallerdifference in contrast between the viewpoint images than in areas wherethere is a larger difference in contrast between the viewpoint images.The image processing for sharpening and smoothing on the viewpointimages is also performed more intensely in areas where the contrastdistribution M_(CON)(j, i) is higher than in areas where the contrastdistribution M_(CON)(j, i) is lower.

In the present exemplary embodiment, Eqs. (2), (3), (4A), and (4B)perform the sharpening processing in areas where the image displacementamount distribution M_(DIS)(j, i) shows a smaller difference from apredetermined shift amount (reference). The smoothing processing isperformed in areas where the image displacement amount distributionM_(DIS)(j, i) has a larger difference. In the present exemplaryembodiment, Eqs. (2), (3), and (4A), perform the sharpening processingmore intensely in areas where the image displacement amount distributionM_(DIS)(j, i) has a smaller difference from, the predetermined shiftamount than in areas where the image displacement amount distributionM_(DIS)(j, i) has a larger difference.

In the present exemplary embodiment, Eqs. (2), (3), and (4B), perform,the smoothing processing more intensely in areas where the imagedisplacement amount distribution M_(DIS)(j, i) has a larger differencefrom, the predetermined shift amount than, in areas where the imagedisplacement amount distribution has a smaller difference.

Further, in the present exemplary embodiment, Eqs. (4A) and (4B),perform the processing for increasing a difference between the pluralityof viewpoint images to sharpen the parallax or reducing a differencebetween the plurality of viewpoint images to smooth the parallax foreach pixel of the viewpoint images, whereby the plurality of modifiedviewpoint images is generated. The first sharpening processing of Eq.(4A) and the first smoothing processing of Eq. (4B) are arithmeticprocessing between the first viewpoint image I₁(j, i) and the secondviewpoint image I₂(j, i) which are the output signals of the first andsecond photoelectric conversion units included in each pixel (j, i).

[Refocusing by Shift Composition Processing]

In step S6 of FIG. 2, the control unit 101 performs processing formultiplying the first and second modified viewpoint images MI₁(j, i) andMI₂(j, i) (first to N_(LF)-th modified viewpoint images) by respectiveweighting factors, relatively shifting the products in the pupildivision direction (x-axis direction), and adding the results (shiftcomposition processing). The control unit 101 thereby generates anintermediate image which is a composite image of the plurality ofviewpoint images.

FIG. 4 is an explanatory diagram illustrating an outline of refocusingby the shift composition processing of the first modified viewpointimage MI₁(j, i) and the second modified viewpoint image MI₂(j, i)(plurality of modified viewpoint images) in the pupil division direction(x-axis direction). In FIG. 4, the x-axis is taken in a verticaldirection of the diagram while downward is defined as the positivedirection of the x-axis as. A y-axis is taken in a directionperpendicular to the diagram while a near side from a drawing surface isdefined as the positive direction of the y-axis. A z-axis is taken in ahorizontal direction of the diagram while leftward is defined as thepositive direction of the z-axis. FIG. 4 schematically illustrates thefirst modified viewpoint image MI₁(j, i) and the second modifiedviewpoint image MI₂(j, i).

The first and second modified viewpoint images MI₁(j, i) and MI₂(j, i)(plurality of modified viewpoint images) contain not only lightintensity distribution information but incident angle information aswell. A refocused image on a virtual imaging plane 610 can thus begenerated by the following parallel translation and addition processing.First, the first modified viewpoint image MI₁(j, i) is paralleltranslated up to the virtual imaging plane 610 along a principal rayangle θ₁. The second modified viewpoint image MI₂(j, i) is paralleltranslated up to the virtual imaging plane 610 along a principal rayangle θ₂. Secondly, the translated first and second modified viewpointimages MI₁(j, i) and MI₂(j, i) are added.

Parallel translating of the first modified viewpoint image MI₁(j, i) upto the virtual imaging plane 610 along the principal ray angle θ₁corresponds to a shift of −1 pixel in the column direction. Paralleltranslating of the second modified viewpoint image MI₂(j, i) up to thevirtual imaging plane 610 along the principal ray angle θ₂ correspondsto a shift of +1 pixel in the column direction. In such a manner, thefirst and second modified viewpoint images MI₁(j, i) and MI₂(j, i) canbe relatively shifted by +2 pixels, and the first modified viewpointimage MI₁(j, i) and the second modified viewpoint image MI₂(j, i+2) canbe associated and added to generate a refocused signal on the virtualimaging plane 610.

In step S6 of FIG. 2, the control unit 101 generates a shift compositeimage I_(S)(j, i), which is the refocused image on the virtual imagingplane 610, from the first and second modified viewpoint images MI₁(j, i)and MI₂(j, i) (plurality of modified viewpoint images) using Eq. (5):

In Eq. (5), pe is an even number closest to the predetermined imagedisplacement amount p. The even number pe closest to the predeterminedimage displacement amount p is calculated by pe=2×ROUND (p/2), whereROUND is a rounding-off function.

I _(s)(j,i)=MI ₁(j,i)+MI ₂(j,i−pe).  (5)

The shift composition processing of the first and second modifiedviewpoint images MI₁(j, i) and MI₂(j, i) is not limited to the evennumber shifting and addition processing. Real number shifting and moregeneric composition processing may be used if needed. The processing ofstep S7 of FIG. 2 to be described below may be omitted according toneed, and the shift composite image I_(S)(j, i) generated by shiftingand adding the first and second modified viewpoint images MI₁(j, i) andMI₂(j, i) (plurality of modified viewpoint images) may be output as anoutput image.

[Sharp/Unsharp Control]

In step S7 of FIG. 2, the control unit 101 performs second sharpeningand second smoothing processing according to the image displacementdifference amount distribution M_(DIFF)(j, i) on the shift compositeimage I_(S)(j, i) (intermediate image) generated from the first andsecond modified viewpoint images MI₁(j, i) and MI₂(j, i) (first toN_(LF)-th modified viewpoint images). By such processing, the controlunit 101 generates an output image that is subjected to sharp/unsharpcontrol for adaptively controlling areas of high sharpness and areas ofhigh degrees of blur after imaging.

In the present exemplary embodiment, the control unit 101 performs thesecond sharpening processing on the shift composite image I_(S)(j, i) inareas where the image displacement difference amount distributionM_(DIFF)(j, i) is 0 or above (M_(DIFF)(j, i)≧0). On the other hand, thecontrol unit 101 performs the second smoothing processing in areas wherethe image displacement difference amount distribution M_(DIFF)(j, i) isbelow 0 (M_(DIFF)(j, i)<0). The control unit 101 thereby generates theoutput image.

In step S7 of FIG. 2, the control unit 101 initially sets a secondintensity parameter K_(USM)≧0 for specifying intensity of the secondsharpening processing or second smoothing processing on the shiftcomposite image I_(S)(j, i).

In step S7, the control unit 101 then applies a two-dimensional low-passfilter {F_(LPF)(j_(LPF), i_(LPF))|−n_(LPF)≦j _(LPF)≦n_(LPF),−m_(LPF)≦i_(LPF)≦m_(LPF)} to the shift composite image I_(S)(j, i). Thecontrol unit 101 calculates an unsharp mask I_(USM)(j, i) using Eq. (6):

For example, a two-dimensional filter such as ^(t)[1, 0, 2, 0, 1]×[1, 0,2, 0, 1] may be used as the two-dimensional low-pass filterF_(LPF)(j_(LPF), i_(LPF)). A two-dimensional Gaussian distribution maybe used if needed.

$\begin{matrix}{{I_{USM}\left( {j,i} \right)} = {{I_{S}\left( {j,i} \right)} - {\sum\limits_{j_{LPF} = {- n_{LPF}}}^{n_{LPF}}{\sum\limits_{i_{LPF} = {- m_{LPF}}}^{m_{LPF}}{{F_{LPF}\left( {j_{LPF},i_{LPF}} \right)} \times {{I_{S}\left( {{j + j_{LPF}},{i + i_{LPF}}} \right)}.}}}}}} & (6)\end{matrix}$

In step S7, the control unit 101 finally applies the unsharp maskI_(USM)(j, i) to the shift composite image I_(S)(j, i) according to theimage displacement difference amount distribution M_(DIFF)(j, i) toperform. the second sharpening or smoothing processing using Eq. (7).The control unit 101 thereby generates a refocused image I_(RF)(j, i)which is the output image.

I _(RF)(j,i)=I_(S)(j,i)+k _(USM) ×M _(DIFF)(j,i)×I _(USM)(j,i).  (7)

In areas where the image displacement difference amount distributionM_(DIFF)(j, i) is 0 or above (M_(DIFF)(j, i)≦0), Eq. (7) represents thesecond sharpening processing. The second sharpening processing isprocessing for sharpening the shift composite image I_(S)(j, i)according to the magnitude of the image displacement difference amountdistribution M_(DIFF)(j, i) using the unsharp mask I_(USM)(j, i)multiplied by a positive coefficient k_(USM)×M_(DIFF)(j, i).

On the other hand, in areas where the image displacement differenceamount distribution M_(DIFF)(j, i) is below 0 (M_(DIFF)(j, i)<0), Eq.(7) represents the second smoothing processing. The second smoothingprocessing is processing for smoothing the shift composite imageI_(S)(j, i) according to the magnitude of the image displacementdifference amount distribution M_(DIFF)(j, i) using the unsharp maskI_(USM)(j, i) multiplied by a negative coefficient k_(USM)×M_(DIFF)(j,i).

The refocusing by the shift composition processing can be performedbased on an optical principle using light field (LF) data. Therefocusing by the shift composition processing has an advantage that theprocessing can be applied even to areas where the image displacementdifference amount distribution M_(DIFF)(j, i) is not detectable. If thex-axis direction (y-axis direction) is the only pupil divisiondirection, like the pupil division of the present exemplary embodiment(Nx=2, Ny=1, and N_(LF)=2), the following situation can occur. That is,a refocusing effect is obtained in the x-axis direction (y-axisdirection) which is the pupil division direction, while a sufficientrefocusing effect is not obtained in the y-axis direction (x-axisdirection) orthogonal to the pupil division direction. On the otherhand, a blur control by the sharpening and smoothing according to theimage displacement difference amount distribution M_(DIFF)(j, i) canprovide a refocusing effect regardless of the pupil division direction.In the present exemplary embodiment, the processing for adjusting theresolution feeling is performed by combining the refocusing by the shiftcomposition processing with the blur control by the sharpening andsmoothing according to the image displacement difference amountdistribution M_(DIFF)(j, i). A refocusing effect can thus be obtainedeven in the direction orthogonal to the pupil division direction.

As described above, in the present exemplary embodiment, the outputimage is generated by performing the image processing for sharpening andsmoothing according to the image displacement amount distributionM_(DIS)(j, i) on the composite image I_(S)(j, i) of the plurality ofmodified viewpoint images. More specifically, Eq. (5) adjusts the shiftamount to adjust the position in the depth direction at which theresolution feeling is the most enhanced. The coefficients k_(ct) andK_(USM) in Eqs. (3) and (7) can also be modified according to useroperations, whereby the intensity of the resolution feeling of theoutput image (display image) can be adjusted.

In the present exemplary embodiment, the adjustment width of theintensity of the resolution feeling is changed according to theInternational Organization for Standardization (ISO) sensitivity whenthe viewpoint images are captured. The reason is that the higher the ISOsensitivity, the lower a signal-to-noise (S/N) ratio, and a strongadjustment applied to the resolution feeling of the image can furtherreduce the image quality. FIG. 6A illustrates a table of examples ofsetting values. The table is stored in the ROM 103 in advance. However,the table may be modified by user settings. Since the shift amountbetween the viewpoint images is unaffected at this time, the adjustmentwidth of the shift amount is constant regardless of the ISO sensitivity.

Next, processing for changing a point of view by controlling thecomposition ratio of the plurality of viewpoint images, which is theprocessing specific to the present exemplary embodiment, will bedescribed. Such processing has an effect of displaying a main objectmore sharply by controlling the composition ratio to change the point ofview and shift a blur. Basically, the plurality of viewpoint images issynthesized in a specified area at a specified composition ratio, and aresultant image is superimposed and displayed on the original compositeimage. A composite image J(j, i) at coordinates (j, i) is expressed byEq. (8):

J(j,i)=(1−k _(view))I ₁(j,i)+k _(view) I ₂(j,i).  (8)

k_(view) can be adjusted to adjust the effect of a blur influenced fromother objects.

In the present exemplary embodiment, the composition ratio of theviewpoint images is manually set. However, a setting manner is notlimited thereto. The composition ratio may be automatically determined.For example, the composition ratio may be variously changed to generatea plurality of composite images and the change in the composite imagesmay be detected to automatically set an optimum composition ratio thatis effective for blur shift.

Next, the ghost reduction processing using a difference in the effect ofghosts obtained from the plurality of viewpoint images, which isprocessing specific to the present exemplary embodiment, will bedescribed. In the present exemplary embodiment, the difference betweenthe plurality of viewpoint images is determined area by area (or pixelby pixel), and only positive difference values are extracted to detectunwanted light components (ghost components). The detected unwantedlight components are subtracted from the image to be displayed orrecorded (in the present exemplary embodiment, the composite image ofthe plurality of viewpoint images) to carry out ghost reductionprocessing. If three or more viewpoint images are relevant, and thereare several difference values (difference images) as a result ofcombination of the viewpoint images, a combination that maximizes (themaximum value or accumulated value of) the differences can be selectedand processed to perform the ghost reduction processing to the maximumeffect.

Some lenses may be characteristically less likely to produce a ghost.Therefore, in the present exemplary embodiment, control is performed toenable or disable the ghost reduction processing based on lensinformation when the viewpoint images are captured. Specifically, if alens is less likely to produce a ghost, control is performed to disablemodification of the setting of the ghost reduction processing. Whether alens is less likely to produce a ghost, or whether the lens needs aghost adjustment, is stored in a table illustrated in FIG. 6B to bedescribed below.

FIG. 5A illustrates an example of a GUI to be applied to the image data,the resolution feeling adjustment processing, the blur shift processingby changing the point of view, and the ghost reduction processing, whichare characteristic of the present exemplary embodiment. A window 500represents a display of the entire application for applying theforegoing processing to the image data. A display image 501 isconfigured to display the image to be processed. When image processingis applied to the display image 501 by using various parameters set onthe application, the display image 501 is updated to display the imagedata subjected to the processing.

A palette 502 is a GUI for setting parameters and application ranges ofimage processing including the image processing characteristic of thepresent exemplary embodiment. In the present exemplary embodiment, theresolution feeling adjustment processing and the blur shift processingcan be simultaneously applied. The ghost reduction processing is to beexclusively applied without carrying out the other processing. However,exemplary embodiments are not limited thereto. Each type of processingmay be configured to be executable only in an exclusive manner so thatthe accuracy of the processing can be ensured and maintained easily. Allthe processing may be configured to be simultaneously applicable toincrease the degree of processing freedom. The palette 502 includes auser interface (UI) for selecting either a resolution feelingadjustment/blur shift or ghost reduction. FIG. 5A illustrates a case inwhich a resolution feeling adjustment/blur shift is selected.

A parameter setting screen for a resolution feeling adjustment displaysthe following: a checkbox for setting whether to perform a resolutionfeeling adjustment, a bar 503 and an index 503 p for adjusting aposition in the depth direction where the resolution feeling ismaximized (adjusting the shift amount between the viewpoint images), anda bar 504 and an index 504 p for adjusting the intensity of theresolution feeling adjustment.

The bar 503 is configured to indicate corresponding directions “far (0)”and “near (0)” at the top and bottom. The accompanying numerals indicatea degree of adjustment (shift) with reference to the center of the bar503. In the present exemplary embodiment, five levels to the far sideand five levels to the near side, or a total of 11 levels, can be set.The number of settable levels of adjustment and the actual shift amountof pixels allocated for one level can be arbitrarily set.

The palette 502 displays “intensity (5)” as the set level of intensitycorresponding to the bar 504. In the present exemplary embodiment, 11levels of intensity can be set in terms of integers of 0 to 10 with theleft end as 0. As described above, k_(ct) and k_(USM) are set tocorrespond to the level of intensity. As the numeral increases, K_(ct)is set to increase the separation performance of the two images andk_(USM) is set to enhance an effect of edge and blur processing.

In a parameter setting screen for a blur shift, followings aredisplayed: a checkbox for setting whether to perform, a blur shift, anda button 505 to enter processing for selecting an area to apply the blurshift to. If an instruction (cursor designation or touch operation) tooperate the button 505 is given, an operation for selecting the area tobe applied within the image is enabled. In the setting screen, acheckbox 506 for setting whether to display a boundary of the selectedarea, and a bar 507 and an index 507 p for setting a composition ratioare also displayed. The bar 507 is configured to indicate “left (0)” and“right (0)” on the left and right, to indicate a direction of the pointof view in which the image generated by the setting of the compositionratio is seen. The accompanying numerals indicate how close the image issynthesized to one point of view with reference to the center of the bar507 (where the viewpoint images are composited at a ratio of 1:1).

In a setting screen for the ghost reduction processing followings aredisplayed: a checkbox for setting whether to perform ghost reduction, abutton 508 for entering processing for selecting an area to apply theghost reduction to, and a checkbox 509 for setting whether to display aboundary of the selected area.

Operations on the foregoing palette 502 and the display image 501 may bemade according to instructions from the input devices such as a keyboardand a mouse. Alternatively, the indexes and buttons may be operated bytouching from a touch panel.

A processing flow for determining adjustment ranges and whether toadjust the parameters of the image processing characteristic of thepresent exemplary embodiment will be described with reference to theflowchart of FIG. 5B. Steps S5010 to S5040 are each executed by thecontrol unit 101 or by various components according to instructionsfrom, the control unit 101. If the user operates the operation, unit105, the control unit 101 activates software based on a program, readsan image file including viewpoint images (viewpoint image file) from theexternal storage device 104, and loads the viewpoint image file into theRAM 103, whereby the processing illustrated in FIG. 5B is started.

In step S5010, the control unit 101 obtains imaging information recordedas metadata from the read out viewpoint image file. In the presentexemplary embodiment, the control unit 101 refers to the ISO sensitivityset at the time of imaging as the imaging information.

In step S5020, the control unit 101 obtains identification (ID)information corresponding to the model of the imaging apparatus,recorded as metadata from the read out viewpoint image file.

In step S5030, the control unit 101 refers to a table such as thatillustrated in FIG. 6A recorded in the ROM 108, and sets an adjustablerange of intensity of the resolution feeling adjustment corresponding tothe ISO sensitivity. Suppose, for example, that the plurality ofviewpoint images (and the composite image) is captured at an ISOsensitivity of 1600. According to FIG. 6A, since 800≦(the ISOsensitivity of the images)≦3200, the corresponding coefficients K_(ct)and k_(USM) are multiplied by a correction coefficient of 0.8,accordingly to a limited effect. In the present exemplary embodiment,the number of possible levels of adjustment indicated by the bar 504 iskept unchanged at 11 levels while the corresponding inside parametersare multiplied by the correction coefficient. However, the exemplaryembodiment is not limited thereto. The number of possible levels ofadjustment of the bar 504 may be reduced to indicate the absoluteintensity of the effect.

In step S5040, the control unit 101 refers to a table such as thatillustrated in FIG. 6B recorded in the ROM 108, and sets theapplicability of the ghost reduction processing according to the lenstype obtained from the metadata. According to the table of FIG. 6B, ifthe lens is the lens A, the optical system is less likely to produce aghost or there occurs only a ghost that cannot be removed under thecurrently-obtained parallax of the viewpoint images. In such a case, asetting is made such that the ghost reduction processing described inthe present exemplary embodiment is not applicable. For example, thesetting portion of the ghost reduction processing on the palette 502 isgrayed out. Alternatively, the application of the ghost reductionprocessing may be cancelled inside the apparatus even if the user checkson the checkbox to apply the ghost reduction processing.

In the present exemplary embodiment, the refocusing intensity slider(the maximum value of the intensity of parallax enhancement) isconfigured to vary only with the ISO sensitivity. However, therefocusing intensity slider may be configured to vary with imagingconditions (imaging information) such as the aperture value (F value)and a focal length of the lens. According to the exemplary embodiment,the width of the intensity of the resolution feeling adjustment isvariable with the ISO sensitivity, and the adjustment width of the shiftamount of the refocus processing is fixed. However, the adjustmentwidths may be either fixed or variable, for example, depending on thecharacteristics of the image sensor and other imaging conditions(aperture value and focal length). The ghost reduction may beimplemented such that an adjustment value can be set even if the lens isless likely to produce a ghost, which is difficult for eyes to notice.

Further, a plurality of viewpoint images obtained by an image sensor inwhich a plurality of pixels is assigned to each single microlens likethe present exemplary embodiment is particularly susceptible to shadingdue to vignetting. Shading correction is in advance performed on each ofthe plurality of viewpoint images. When a correction function fordetecting shading occurring in each viewpoint image is detected from,the pixel values of the viewpoint image, a correct shading function maynot be detected due to pixels of which signals are saturated by incidentlight or pixels to which charges leak in from around. In such a case,since the shading correction, cannot be properly performed, the shadingcorrection is set disabled. Viewpoint images lacking proper shadingcorrection may not be subjected to appropriate blur shift processing orghost reduction processing. Therefore, information about whether theshading correction has been successful or information indicating whetherthe shading correction is enabled or disabled (in which case the shadingcorrection is yet to be applied to the image) is recorded in themetadata of the image file as imaging information. On the setting screenfor making settings of the blur shift processing or the ghost reductionprocessing, the control unit 101 imposes a limitation not to apply theprocessing based on the information that the shading processing has notbeen successful or that the shading processing is disabled. To imposesuch a limitation, a setting may be made such that the checkboxes forthe blur shift processing and the ghost reduction processing may begrayed out and disabled. Alternatively, the processing may be madeinapplicable (disabled) regardless of the positions of the indexes evenif the user checks on the checkboxes and moves the indexes.

The exemplary embodiments of the present exemplary embodiment have beendescribed in detail above. However, the present disclosure is notlimited to such exemplary embodiments, and various modes not departingfrom the gist of the disclosure are also covered by the presentdisclosure. Parts of the foregoing exemplary embodiments may beappropriately combined. An exemplary embodiment of the presentdisclosure may include supplying a software program for implementing thefunctions of the foregoing exemplary embodiments to a system or anapparatus including a computer that can execute the program, directlyvia a recording medium or by using wired/wireless communications.Therefore, a program code itself that is supplied to and installed onthe computer to implement the functional processing of the presentdisclosure also constitutes an exemplary embodiment of the presentdisclosure. In other words, a computer program itself for implementingthe functional processing of the present disclosure is also covered bythe present disclosure. In such a case, the program may be in any modeas long as the program has such functions. Examples of the mode of theprogram include object code, a program to be executed by an interpreter,and script data supplied to an operating system (OS). Examples of therecording medium for supplying the program may include a magneticrecording medium such as a hard disk and a magnetic tape, anoptical/magnetooptical recording media, and a nonvolatile semiconductormemory. Examples of the method for supplying the program may includestoring a computer program constituting an exemplary embodiment of thepresent disclosure in a server present on a computer network, anddownloading and installing the computer program by a connected clientcomputer.

An image processing apparatus for performing image processing using aplurality of viewpoint images and a method for controlling the sameprovide a desired image effect by a user's simple operations.

OTHER EMBODIMENTS

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from, a network or thestorage medium. The storage medium, may include, for example, one ormore of a hard disk, a random-access memory (RAM), a read only memory(ROM), a storage of distributed computing systems, an optical disk (suchas a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc(BD)™), a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, the scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2016-112159, filed Jun. 3, 2016 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: an acquisition unit configured to obtain a plurality of viewpoint images and imaging information corresponding to the plurality of viewpoint images; an image processing unit configured to apply image processing to image data based on the viewpoint images; a setting unit configured to set a parameter of the image processing by the image processing unit based on a user operation; and a limitation unit configured to limit the parameter settable by a user via the setting unit in the image processing based on the imaging information.
 2. The image processing apparatus according to claim 1, wherein the imaging information includes ISO sensitivity, a focal length, a lens, and an aperture value when the viewpoint images are captured.
 3. The image processing apparatus according to claim 1, wherein a parameter of adjustment processing for adjusting resolution feeling of an image is intensity of processing, the adjustment processing being applied by the image processing unit, the parameter being set by the setting unit.
 4. The image processing apparatus according to claim 1, wherein a parameter of blur shift processing for shifting a position of a blur in an image is a composition ratio in synthesizing the viewpoint images, the blur shift processing being applied by the image processing unit, the parameter being set by the setting unit.
 5. The image processing apparatus according to claim 1, wherein the viewpoint images are images obtained from an image sensor in which a plurality of photoelectric conversion elements is assigned to a microlens.
 6. The image processing apparatus according to claim 1, wherein the viewpoint images are images obtained from a multi-lens imaging apparatus.
 7. The image processing apparatus according to claim 1, wherein an adjustment width of intensity of the processing for adjusting resolution feeling of an image is variable by the limitation unit based on the imaging information, the adjustment processing being applied by the image processing unit, the intensity being set by the setting unit.
 8. The image processing apparatus according to claim 1, wherein an adjustment width of intensity of the processing for adjusting resolution feeling of am image is fixed regardless of the imaging information, the adjustment processing being applied by the image processing unit, the intensity being set by the setting unit.
 9. The image processing apparatus according to claim 1, wherein an adjustment width of a shift amount of the viewpoint images in blur shift processing for shifting a position of a blur in an image is variable by the limitation unit based on the imaging information, the blur shift processing being applied by the image processing unit, the shift amount being set by the setting unit.
 10. The image processing apparatus according to claim 1, wherein an adjustment width of a shift amount of the viewpoint images in blur shift processing for shifting a position of a blur in an image is fixed regardless of the imaging information, the blur shift processing being applied by the image processing unit.
 11. The image processing apparatus according to claim 1, wherein processing for reducing a ghost within an image is switched between an enabled and disabled state based on the imaging information, the ghost reduction processing being applied by the image processing unit.
 12. The image processing apparatus according to claim 11, the imaging information based on which the ghost reduction processing is switched between the enabled and disabled state is lens information when the viewpoint images are captured.
 13. The image processing apparatus according to claim 1, the processing for reducing a ghost in an image is constantly in an enabled state regardless of the imaging information, the ghost reduction processing being applied by the image processing unit.
 14. A method for controlling an image processing apparatus, comprising: obtaining a plurality of viewpoint images and imaging information corresponding to the plurality of viewpoint images; imposing a limitation on a user-settable parameter in image processing by an image processing unit based on the imaging information; setting a parameter of the image processing by the image processing unit based on a user operation and the limitation; and applying the image processing by the image processing unit to image data based on the viewpoint images. 